Tomáš Pospíšek's Notizblock
Lately I had to script some failover procedures.
We needed to switch this service to a different environment, to rewrite that config file to point to a different ressource and restart it etc.
At every step that was taken, the question was: "what if this step would fail?". Continuing after a failure of any single step would transition the system into an even more incoherent state.
Determining the reason for the failure and trying to fix it seemed not trivialy feasable. Rolling back wasn't possible either because the reason to failing over is often enough that the system as a whole is useless in its current state.
Since the answer to the above question was hard, i.e. we would not know why some step would have failed, the answer that we chose was: "Abort and let the admin decide on how to proceed".
The script started as a collection of one liners that configured various aspects of the system pushing it over to it's failovered state.
Common sequences of steps were combined into shell functions, utility and helper routines were written resulting in a script that wasn't a simple sequence of shell commands any more.
So provided that there would be some problem, as would be expected in the case of a failover, the admin would have a very hard time following the various steps the script would have taken, needing to follow the script through various functions and decision forks.
Miserably I decided to revert the script back to more or less a list of one liners plus very sparse convenience functions.
Concentrating repetive tasks into subroutines and removing redundancies is one of the first, fundamental paradigms of programming and still, here is a case where doing just that would be detrimental.
Having a good editor, that has a good search and replace function seems to be good enough to change parameters of the script (such as host names) in case the system has to be modified for new hardware or scaled to more machines.
Once again it seems, that there is no one size fits all in the world of programming and that, in order to find good solutions one is forced to reflect on the "why"s and the adequacy of even the most basic programming rules.
Quite desillusioning that it seems to be so :-|
Tomáš Pospíšek, 2009-10-27
